Regularization versus Early Stopping: a Case Study with a Real System

Regularization and Early Stopping are two of the most common techniques to deal with the overtraining problem in the Artificial Neural Networks field. The overtraining problem appears mostly in systems affected by noise in which after a certain amount of training, the neural network used for modelling starts to learn information specific from the training signal or the noise. It has already been shown that these techniques can be used to avoid this problem and they are formerly equivalent, but this issue deserve further investigation since real systems sometimes behave in a different way than simulated systems. A fair comparison for the two techniques is not very easy to make since in the networks there are several parameters that cannot be determined in an analytical way. To overcome this difficulty in the present work a procedure for automating the construction of the models has been used. This procedure allows creating models that are optimised in the number of inputs, the number of hidden neurons and the generalization capability using either Early Stopping or Regularization. This enables the possibility of performing a fair comparison. The procedure includes an hybrid direct/specialized training solution for evaluating the inverse model. To test the results the system used is a reduced scale prototype kiln affected by measurement noise, for which the Direct Inverse Control and Internal Model Control strategies were implemented.